ci(test): add S3 integration gate to the merge-queue test workflow#915
Draft
mbrobbel wants to merge 3 commits into
Draft
ci(test): add S3 integration gate to the merge-queue test workflow#915mbrobbel wants to merge 3 commits into
mbrobbel wants to merge 3 commits into
Conversation
Wire the MinIO-backed S3 integration tests into the Test workflow as inline steps on the CUDA-13 test-run job. - `make s3-test` runs `[s3][integration]~[large]~[aws]` (standard correctness gate; testcontainers brings up HTTP + TLS MinIO automatically via SIRIUS_TEST_S3_AUTO=1 set by the Makefile target). - `make s3-test-large` runs the large SF10 SQL-over-S3 suite (`[s3][sql][large]`) with SIRIUS_TEST_S3_LARGE=1 so the harness generates and uploads lineitem_sf10.parquet. - A `docker info` diagnostic step runs first so runner capability is visible in the job log. Steps are gated to `matrix.cuda.version == '13'` (phase 1: runs on PRs carrying this branch so Docker availability on the runner can be verified under a draft PR). Before merging, each `if:` will be prefixed with `github.event_name == 'merge_group' &&` so PR CI stays Docker-free and only the merge queue pays the MinIO cost (per the issue proposal). The test-run job timeout is bumped from 60 to 90 minutes to accommodate SF10 fixture generation on top of the existing 45-minute unit-test and TPC-H budgets. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
The gpu-2xl4 runner has Docker installed but the daemon is not running by default. Add a step that calls `sudo systemctl start docker` before the S3 gate steps so testcontainers can reach the Docker socket. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
sudo is not available without a password on the gpu-2xl4 runners, so starting the daemon from the workflow is not feasible. Replace the step with a plain `docker info` that fails fast and clearly until the runner infra is configured to run the Docker daemon on boot. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The MinIO-backed
[s3]integration suite currently runs only manually (make s3-test). A bug like the one in #889 (single-GPU + S3 over MinIO) can slip through PR CI undetected. This PR wires the suite into the merge-queue Test workflow so issues surface before they land ondev.Closes #904.
What this changes
Adds three inline steps to the existing
test-runjob in.github/workflows/test.yml, placed after the TPC-H artifact upload:docker infodiagnostic; warns (never fails) so runner capability is visible in the job log.make s3-test— runs[s3][integration]~[large]~[aws]with testcontainers auto-managing HTTP + TLS MinIO (SIRIUS_TEST_S3_AUTO=1 SIRIUS_TEST_S3_STRICT=1).make s3-test-large— runs the SF10 SQL-over-S3 suite ([s3][sql][large]) withSIRIUS_TEST_S3_LARGE=1so the harness generates and uploadslineitem_sf10.parquet.The test-run job timeout is bumped from 60 → 90 minutes to cover SF10 fixture generation on top of the existing 45-minute unit-test and TPC-H budgets.
No build-job changes:
build.taralready contains the S3-capable binary andduckdbCLI (the Test workflow is green ondevat the #906 merge commit).Rollout: two phases
Phase 1 (this draft PR): steps are gated to
matrix.cuda.version == '13'and run on any PR carrying this workflow change. Goal: verify Docker is available on thegpu-2xl4self-hosted runner.Phase 2 (before marking ready): prepend
github.event_name == 'merge_group' &&to eachif:condition so:Depends on
#906 (already merged to
dev) — self-managing MinIO via testcontainers.🤖 Generated with Claude Code